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ABSTRACT 


This thesis proposes and analyzes an ad hoc Bayesian method for 
determining lower confidence limits for estimates of mission reliability 
of single components using pass/fail test data. Several test sequences 
with different underlying reliabilities are simulated. After each trial 
the lower confidence limits are calculated using the Classical method, 
the Bayesian method and the proposed method. This is followed by an 
accuracy simulation to estimate the accuracy of the proposed method. 

The proposed method shows promise for future usefulness in that it is 
capable of demonstrating a high degree of reliability with a relatively 
small number of trials while maintaining some degree of protection 


against accepting unreliable components. 
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I. INTRODUCTION 


Using pass/fail test data, there are only two methods available for 
determining lower confidence limits for estimates of mission reliability 
of single components. The methods are the Classical method and Bayesian 
method, 

The Classical method considers mission reliability as a constant, 
and calculates the lower confidence limit by solving for the parameter 
p in the Binomial Distribution, B(x;n,p). The basic advantage to this 
method is that it provides a conservative estimate of mission reliability, 
providing some protection against accepting unreliable equipment. The 
basic disadvantage of the Classical method is that one must pay a high 
premium for this conservative estimate. This premium is in the form 
of requiring a large number of trials in order to demonstrate a high 
degree of reliability. Even if no failures occur in 30 trials, the 5% 
lower confidence limit using the Classical method is only .91. With 
developing weapon systems becoming more complex and hence more expensive, 
determination of mission reliability using Classical methods will become 
more and more economically infeasible. 

The Bayesian concept considers mission reliability, R, as a random 
variable. One particular Bayesian method assumes R has a Beta distribution. 
Prior to testing, a Beta density (prior density) is assumed and the test 
data is then used to modify the parameters of the prior density, resulting 
in a posterior density. The lower confidence limit is then calculated 
by determining the appropriate percentile point of this posterior Beta 
density. One advantage to this method is that it allows the use of past 
mission test data of technically similar systems in the determination of 


the reliability of the system being tested. Engineers and scientists 





who are designing the present system and have had experience with 
technically similar systems, generally have sound intuition concerning 
the mission reliability of the present system, The greatest advantage 
of the Bayesian method is that by utilizing this intuition a high degree 
of reliability can be demonstrated with a relatively small number of 
trials. The greatest shortcoming of the Bayesian method occurs when 
the above intuition is incorrect, When the sum of the Beta parameters 
(a + b) in the prior density is much greater than the number of trials 
to be observed, then the resulting lower confidence level is more a 
function of the assumed prior density than of the trials results. Thus, 
when the assumed prior density is incorrect and the sum of its parameters 
is much greater than the number of trials to be observed, then the 
resulting lower confidence level will be misleading. 

With emerging systems becoming more and more costly, the need for 
a method to determine, with a small number of trials, accurate lower 
confidence limits of estimated reliability will become more and more 
pressing. The Bayesian method holds promise in this direction; but 
with no protection against an incorrect prior density, this method 
leaves much to be desired. Clearly then a Bayesian method, modified 
to provide protection against an incorrect prior density, would not be 
without merit. It was this thought that motivated the research detailed 


in this thesis. 





II, SUMMARY 


An ad hoc modified Bayesian technique was devised for calculating 
100(1-a)% lower confidence limits, This technique has the capability 
of demonstrating high degrees of reliability with a relatively small 
number of trials, while maintaining some degree of protection against 
accepting inferior components, While the proposed method does show 
promise for future usefullness, more analysis needs to be done before 
mes Pub to use, 

Prior to the testing phase, the government and the contractor will 
jointly decide on an initial prior Beta density along with a positive 
integer value for k. The proposed method uses Bayesian techniques, 
with the only exception being that the prior density is subject to 
change. As long as no failures occur, the initial prior density is 
used, Once a failure occurs the prior density is changed so that its 
mean is equal to the maximum likelihood estimate of the reliability at 
that point in the test sequence, If after a failure, k successive 
successes occur, the prior density is adjusted further towards the 
imitial= prior and so on. 

Several test sequences with different underlying reliabilities 
were simulated. After each trail the 100(1-~)% lower confidence limit 
was calculated using the Classical method, the Bayesian method and the 
proposed method, After this an accuracy simulation was conducted to 
estimate the accuracy of the proposed method. As stated above, the 


results were promising. 





Pirie AALS IS 


In order to combine the advantages of both the Classical and 
Bayesian methods, the following ad hoc modified Bayesian technique 
was proposed for determining the lower 1C00(1-7)% confidence limit of 
the mission reliability of a single component, R, (2). 

The government and the contractor will jointly decide cn an initial 
prior density for mission reliability, which is 3(xj3a,b0); by cetermining 
a value for a 70 and bo70. In addition to this prior density they will 
also decide on a value for k, which is a positive integer. A number 
of components would then be tested and given a chance to live out one 
mission, At any point during the test sequence, standard Bayesian 
techniques will be used to calculate Ry (a) s except the prior density 
used in the calculation will be varied as follows. 

a) If no failures have occurred then the prior density is 
just the predetermined prior density, g(r3a,,d,). 

b) If i failures have occurred then the prior is B(r3a,,b,), 
where a. and be are determined as follows: let n denote 
the number of tests to data, and n. denote the test number 
on which the ee failure occurred, then 


if OSn—-n.<k then Rr) A Rr) and 5 eR 
2 eee a ee 


(0) 


1 
sia ksn-n, <2k then a,cal laa 1a and b,=b! ey +o) 


2 2 
2) ee) a2) oe) 
at eksn-n,<5k then a.=a; =a ta; and b,=b; al any 
2 D 
ee 5 ion 3) Gat) 
ae jksn-n, <(j+1)k then aca Jaa sald ) and b,=b! =b tb; 
2 2 


OTM) 9544) 5.0 0s 
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Once the prior density has been determined, then the parameters a and b 
of the posterior density are determined by adding the number of successes 
to date, to Aes and adding the number of failures to date, to b,. in 
mathematical terms a=a,+n-i and b=b,+i. The 100(1-%)% confidence limit, 
R, (2), is then determined by finding the a= percentile point of the 
posterior density, 8(r;a,b). Since the Beta density is not defined for 
a or b equal to zero, then R, (2) is taken to the identically equal to 
zero, then R, (¢) is taken to be identically equal to zero wntil the 
first success has been achieved. 

Simulation was the basic technique used to analyze and evaluate the 
accuracy of the method described above. Since the computations required 
in the simulation were completely intractable by hand, the simulation 
was conducted on an IBM 360 computer utilizing the FORTRAN IV source 
language. During the research and development phase of the development 
of a system, it is often the case that the reliability of a component 
is improved as inadequacies are found during initial testing. With 
this in mind, three sequences of test trials were generated, each 
containing increases in their underlying reliability. As in actual 
practice, these increases in the underlying reliability took place 
only after a trial had resulted in a failure. sach test sequence was 
continued until five failures has occurred. The underlying reliabilities 
used to determine the iesult of each trial in the three sequences are 


shown in Table I below. 
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TABLE I 


UNDERLYING RELIABILITIES AFTER HACH FAILURE 


oo By) Ray B3) Bg) 
1 450 20 450 310 80 
Z 10 210 ae ANSTO, 290 
» Aste, 290 oD 095 Als) 


The result of each trial was determined by comparing a uniformily 
distributed random number, on the interval Hora to the appropriate 
reliability shown above, If the random number was less than or equal 
to the underlying reliability, the trial resulted in a success, If 
the random number was greater than the underlying reliability, the 
imialeresulted in a failure. 

After the result of each trial in all the sequences had been determined, 
the 100(1-7)% lower confidence limit was determined after each trial, 
using the Classical method, the Bayesian method, and the proposed ad hoc 
modified Bayesian method. 

The classical 100(1-~)% lower confidence level was determined by 
solving for the parameter p in the Binomial distribution, B(n,p); that 


is by solving for p in the equation 


s ; p ik 
w=) (G)p%(a-p)?9 = f 8(x3s,n-s+2)ax 
O 


J=s 


where © is the significance level, s is the number of successes to date, 


and n is the number of trials to date. Thus the classical 100(1-a)% 


lowen, D. B., Handbook of Statistical Tables, p. 264, Addison-Wesely, 
1962 
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lower confidence level was calculated by finding the ao percentile point 
of the Beta density with parameters; a = the number of successes to date, 
and b = the number of failures to date plus 1. 

To calculate the Bayesian 100(1-a0)% lower confidence limit, the 
results of all previous trials in a sequence were used to modify the 
assumed prior density to determine a posterior density. The Bayesian 
confidence limit was then determined by finding the Aae percentile point 


of the posterior density by solving for R, (a) faethe equation, 
R 
a= i ENG) ets. btn-s )dx 


where a is the value of the parameter a in the prior density, and b is 
the value of the parameter b in the prior density. 

The ad hoc modified Bayesian 100(1-~)% lower confidence limit was 
calculated using the procedures detailed earlier in this section, 

In all of the above three methods, the 100(1-a)% lower confidence 
limit was calculated by determing the Be 8 percentile point of a Beta 
density, with the only difference being in the parameters of the 


densities used. The standardized Beta density function is defined by 


B(xsa,b) = ret x2" (4-x)°"! for 0<x<1,a>0,b>0 


= 0 otherwise 


thus to find the oth percentile point of a Beta density, one had to 


determine the value of the constant [(a + b P 
tase tb 


The basic property used to determine the value of a Gamma function 
was that T(y) = (y-1) T (y-1). This coupled with the approximation 
(accurate to the nearest 1/100,000) that T(1+y) = 1.0 - .5771017y 
+ 9858540y* - ,8764218y° + 832821 2y4 = 5684729y° + 2548205y° 


T 


- .0514993y° for 05y<1, was all that was necessary to evaluate any 


13 





Gamma function, by using a recursive multiplication technique. Due to 
a limitation on the size of a number that can be stored by the computer, 
it was necessary to make the calculations using natural logarithms, 


requiring the addition of logarithms as opposed to the multiplication 


of the actual numbers, The final value for lCatb vas taken as the 
a: Ie - 


antilogerithm of lnLT(atb)] - 1nlT(a)] - 1n[T(b)]. For complete details, 
see the computer program. 


After having determined the value of the constant [T(atb) it was 
T(a)T(d 


then necessary to evaluate the integral pE@y eS x21 (4x) P15, 
O T'(a) saa 


Since the integration would only take place over the lower tail of the 
Beta density, and since the lower tail of the Beta density is reasonably 
flat, a rectangular approximation was used to evaluate the integral. 

To check the accuracy of the above procedures for determining 
percentile points of the Beta density, the procedures were used to 
determine the Pe percentile point for the following values of a and b; 
Cian o. 90, 600, and bai, 1.5,°2.5, 5, 10. The resulting values 
never varied more .01 from the tabulated values, 

In order to gain further insight into the sensitivity of the proposed 
method to the parameter values assumed in the prior density, four pairs 


of parametric values were used, as shown in Table II below. 


TABLS Il 
PRIOR DENSITIES 
a, 45.0 45.0 90.0 90.0 
b 265 520 pre LOR) 


100(1-@)% lower confidence levels were calculated after each trial, using 
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the Classical method, the Bayesian method and ad hoc modified Bayesian 
method with the four different priors shown above and for the modified 
Bayesian method with k=5,10. These lower confidence levels were cal- 
culated for significance levels of .05, .10, and .20. For computed 
results see the computer output. 

After completing this, a simulation was conducted to determine the 
accuracy of the proposed method. To accomplish this, three test 
sequences, each with one underlying reliability and consisting of 40 
trials, were generated. The underlying reliabilities used in the three 
sequences to determine the outcome of a trial were; .80, .90, and .95. 
After each trial, the 100(1-c)% lower confidence limit was calculated 
using the ad hoc modified Bayesian method with the four prior densities 
shown in Table I and for the three significance levels of .05, .10, and 
-cO. The above was then replicated 40 times, resulting in 40 lower 
confidence limits for each trial in each sequence, for each prior density 
and for each value of a, These 40 lower confidence limits were then 
ordered MW sudite to magnitude, If the proposed method is accurate, 
then one would expect 100(1-2)% of the values for each trial to be less 
than or equal to the underlying reliability and 100(%)% of the values 
to be greater than the underlying reliability. For the case where @%=,20, 
and with 40 values, one would expect the ee highest value to be less 
than or equal to the underlying reliability, and one would expect the 
aoe highest value to be greater than the underlying reliability. These 
values are tabulated in Table IV, for the case k=5. As an example, to 
read the accuracy estimate for the proposed method after 25 trials when 


the underlying reliability is .80 and for the prior distribution defined 
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by 2 = 45 and b = 5, one must turn to that portion of the table labeled 
Underlying Reliability = .80, and read down the first column to trial 
number 25, and from columns 2 and 3 (corresponding to a=45, b=5) extract 


the values ,81 and .82, 
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IV. RESULTS 


For a complete listing of the computed results of the first 
simulation, see the Computer Output. This simulation run, along with 
Many additional runs, using different random numbers, pointed out three 
basic characteristics of the proposed method, 

The first characteristic is that the proposed method is cavable of 
demonstrating a high degree of reliability with a relatively small 
number of trials, 

The second and most important characteristic is that the proposed 
method is much more sensitive than the Bayesian method to the underlying 
reliability, thus providing a degree of protection against the assumption 
of an incorrect prior density. For example, in one simulation run the 
first 18 trials resulted in 13 successes and 5 failures. The maximun 
likelihood estimate of the wnderlying reliability would be 13/18 = .72. 
At the end of the 18 trials, the computed 80% lower confidence limits 


were as shown in Table III below, 


TABLE LIT 


80% LOWER CONFIDENCE LIMITS 


A B CCL20 BCL20 WCL20 
(Classical) (Bayesian) (Modified) 
Zone 50 . 60 noc Ae oy 
oso 10.0 .60 ZO) Fed 
4149 or 60 . 86 nod 
St, Del ~60 oe coy 


As pointed out by Sequence 1 in the Accuracy Simulation, the degree of 
protection is not complete, but one must expect to pay some price for 
the capability of demonstrating high degrees of reliability with a small 


number of trials. 
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The third characteristic indicated in this study is that there is 
little difference between the computed ad hoc confidence limit for k=5 
and k=10, The only time the results are greatly different is when the 
trial is the 5th, 6th, 7th, 8th, or 9th consecutive success. Only a 
small portion of the trials fell in the above category. 

For a partial listing of the results from the accuracy simulation 
see Table IV. Sequence 1 of the accuracy simulation demonstrates the 
degree of protection provided by the proposed method. There was a 
significant drop in the reliability estimate between the 7th and 8tn 
trial. Of the 40 replications of Sequence 1, 9 of the replications 
resulted in the first 7 trials being all successes, where only 4 of 
the replications resulted in the first 8 trials being all successes, 
With this being the case, the accuracy estimate for trials 1 thru 7 is 
really an accuracy estimate of the Bayesian method, while for trials 
8 thru 30 it is an accuracy estimate of the ad hoc method, 

Further analysis of the data pointed out two counterintuitive 
characteristics of the proposed method. From the description of the 
proposed method, one would expect the proposed method to yield a result 
Somewhere in between the results yielded by the Classical method and the 
Bayesian method, This is not always the case, If the Bayesian prior 
density underestimates the underlying reliability, then it is possible 
for the ad hoc method to yield a result that is higher than those yielded 
by either the Classical or Bayesian methods, This is caused by the fact 
that the Bayesian method is fairly insensitive to the underlying 
reliability coupled with the fact that the Classical method is very 
conservative when the number of trials is small. For an example of the 


above, see Sequence 3 in the Computer Output. It is also possible for 
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the proposed method to yield a result that is lower than either the 
Classical or the Bayesian result. This result is achieved only when a 
sequence begins with two or more failures, an unlikely event for a 
component having an underlying reliability of .80 or higher. Ifa 
sequence does begin with two failures and then a success, the proposed 
method will continue to give lower results than the Classical method 
until the next failure occurs. 

The second counterintuitive characteristic of the proposed method 
is that it is possible to get an increase in the lower confidence limit 
from trial i to trial i+1, even though trial i+1 resulted in a failure. 
In order for this to occur, with the values of k tested, two conditions 
must hold, The first is that the sequence must begin with a failure, 
followed by a success, and the second condition is that the second 
failure must occur before there are k successive successes. To understand 
what causes the increase in the lower confidence limit it is necessary 
to realize that when a sequence begins as stated above, the proposed 
method will yield the same results as the Classical method until either 
the kth successive success or until the second failure, whichever comes 
first, With this being the case, then if j<k successive successes have 
occurred, the ad hoc posterior density will be 8(x;j,1), and if the 
next trial resuits in a failure the ad hoc posterior density will be 


6(x;25,2) yielding a higher lower confidence limit. 
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TABLE IV 
RESULTS OF ACCURACY SIMULATION FOR w=,20 


UNDERLYING RELIABILITY = .80 


45.0 90.0 A725 
5.0 10.0 Da5 
9th 8th 9th 8th 9th 8th 
Ren beeor 88 .88 93 = 93 
88 88 68 ,.88 94 94 
88 88 88  .88 94 494 
688 .88 89 89 94 294 
68 ,88 89 = 89 94 =. 94 
89 «89 e89 289 94 94 
oo ° 89 89 o89 094 Sst 
82 =, 82 noe G2 HS Mes 
80 83 280 .83 84 .84 
81 81 .80 80 84. .84 
281 81 peo OY 81 85 .85 
a 82 81 81 86 ,86 
80 80 80 ~,80 86 86 
81 83 80 82 087 ~— 88 
81 82 81 82 87 ~~. BT 
82 .82 22 ose 685 =. BT 
83 84 83 =. 84 neue leas 
84 =. B 84 3.84 86 .88 
82 83 82 =, 83 86 =. 86 
82 .82 “ei 282 86 87 
2 Ee 82 .83 86 87 
82 82 nee, Weee Bese) Sei 
282 =, 83 hee | | 2465 285 ~— 85 
81 83 281 82 286 =. 86 
81 82 81 81 285) ~—«. 86 
(ele Ee Nee eee 86 =. 87 
283 83 e262 meT. BT 
083 2.83 82 = 83 POuR 267 
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TABLE IV (Con't) 
RESULTS OF ACCURACY SIMULATION FOR a=,20 


UNDERLYING R&LIABILITY = .90 


45.0 90.0 47.5 

520 10,0 2e0 
9th 8th 9th 8th 9th 8th 
o87 °87 88 2 88 2o 22 
88 88 o 88 - 88 94 94 
88 88 e888 «88 094 294 
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090 90 © 90 © 90 75 eS) 
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71 oh 90 030 2 09) 
091 91 20 © 90 093 99 
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TABLE IV (Con't) 
RESULTS OF ACCURACY SIMULATION FOR a=. 20 


UNDERLYING RELIABILITY = .95 
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V. CONCLUSIONS 


The ad hoc modified Bayesian technique for determining 100(1-a)% 
lower confidence limits, holds significant promise for future usefulness, 
It has the capability of demonstrating high degrees of reliability with 
a relatively small number of trials while maintaining some degree of 
protection against accepting an inferior component because of the 
assumption of a prior density that overestimates the reliability of the 
component, 

While the proposed method shows promise, much analysis still needs 
to be done before it is put to use, More analysis is needed to determine 
the sensitivity of the method to the value of k. The method also has 
to be investigated over a wider range of underlying reliabilities and 
assumed prior densities, In this "Missile Age" where component mission 
reliabilities of .99+ are not rare, a definite need for analysis exists, 
to determine appropriate values for k, After 4xk successive successes 
have occurred, the proposed method yields essentially the same results 
as the Bayesian method, thus to choose a k that is too small will result 
in the loss of protection afforded by the method, To pick a k that is 
too large will result in values quite close to those given by the 


Classical method, thus requiring a larger number of trials, 
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DEFINITION OF VARIABLES IN COMPUTER PROGRAM 


VARIABLE DEFINITION 

A(MM) Value of parameter in a prior Beta density 8(x;a,b) 
for case MM 

AB a for Bayesian prior B(x;a,b) 

ABC AC + BO 

ABG Recurisive value of a when calculating I(a) 

AWT Initial value for a when calculating ad hoc lower 
confidence limits 

AP Value of a when computing a new prior density for 
the ad hoc method after a failure 

AC R * 4 version of IAC 

B(MM) Value of parameter b in prior Beta density B(x3a,b) 
for case Mi 

BB b for Bayesian prior B(x;a,b) 

BBG Recursive value of b when calculating [(b) 

BC R * 4 version of IBC 

BCLO5 Bayesian lower confidence limit for a=,05 

BCL10 Bayesian lower confidence limit for a=,10 

BCL20 Bayesian lower confidence limit for a=,20 

IBle Value of b when computing a new prior censity for 
the ad hoc method after a feilure 

BWI Initial value for b when celculating ad hoc lower 
confidence limits 

CCLO5 Classical lower confidence limit for a=.05 

CCLiI0 Classical lower confidence limit for a=, 10 

CCL20 Classical lower confidence limit for a=.20 

FAABB COmemmauuion Of fractional vortion for calculating 


r(atb) 


D4 





VARIABLE 


FABG 


FBBG 


DEFINITION 


Contribution of fractional portion for calculating 


Pa) 


Contribution of fractional portion for calculating 
Mb) 


In [r(a)] 

Ln [r(arb)] 
r(atb)/T(a) rb) 
r(a) 

T(atb) 

r(b) 

Ln [r(b)] 


Number of failures + 1 to date when generating test 
sequence 


meno Lassicall s(c5a,b) 
b for Classical 8(x;a,b) 


Number of failures to date when calculating ad hoc 
lower confidence limit 


Trial number 
Number of successes since the last failure 


Logical switch to determine if simplified calculation 
can be made to determine GAM 


Dimension counter for cases in ad hoc lower conficence 
limit 


Case number for sequence generation 

Case number for initial values of a and b 
Number of trials in the Mth sequence 
Underlying reliability 


Random number uniform on [0,1] 


2 








VARIABLEn 


T(M,J) 


WCLO5 
WCL10 


WCL20 


DEFINITION 


Logical variable denoting result of Jth trial in 
ith sequence | 


Ad hoc lower confidence limit, @=.05 
Ad hoc lower confidence limit, a=,10 


Ad hoc lower confidence limit, ?=,20 
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